1. Field
This disclosure relates to data quality assurance, and more particularly to a system and method of monitoring and measuring quality of data processed by one or more data processing systems.
2. General Background
Computer networks are providing the ability to interconnect data processing systems so that the data output by one data processing system, an upstream system, can become the input to another data processing system, the downstream system. The downstream system is therefore dependent on the upstream system to produce quality data. One example of such a downstream system is a data warehouse, which captures and stores data from many different sources. A data warehouse becomes a repository of data, from which data can be retrieved for use by any number of downstream systems. In this example, if the data warehouse receives and stores erroneous data, the erroneous data can be proliferated to any number of downstream systems unaware of the questionable quality of the data. Typically, such an error is detected by an end user of the downstream system. This causes the end user to doubt the output produced by the downstream system.
Further and while a downstream system may be operating according to its specifications, it can still produce erroneous output due to the erroneous input data received from an upstream system. Conventionally, a downstream system may include some processing logic designed to check for errors resulting from its own processing. However, since the error was introduced by an upstream system and not by the downstream system's processing, the conventional downstream system's error checking will most likely be inadequate, which will cause error to go undetected by the downstream system.
Thus, it would be beneficial to be able to have reliance on the validity and accuracy of input data used by a data processing system, such as the input data received from an upstream system. In addition, it would be beneficial to be able to detect errors in the input data before it is processed by a downstream data processing system. Further still, it would be beneficial for a data processing system, such as an upstream system, to provide data of a quality which can be used by other data processing system. It would therefore be beneficial to be able to monitor and measure the quality of data input to and output by a data processing system.